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Winner-Take-All (WTA) networks are recurrently connected populations of excitatory and 
inhibitory neurons that represent promising candidate microcircuits for implementing 
cortical computation. WTAs can perform powerful computations, ranging from 
signal-restoration to state-dependent processing. However, such networks require 
fine-tuned connectivity parameters to keep the network dynamics within stable operating 
regimes. In this article, we show how such stability can emerge autonomously through an 
interaction of biologically plausible plasticity mechanisms that operate simultaneously on 
all excitatory and inhibitory synapses of the network. A weight-dependent plasticity rule 
is derived from the triplet spike-timing dependent plasticity model, and its stabilization 
properties in the mean-field case are analyzed using contraction theory. Our main result 
provides simple constraints on the plasticity rule parameters, rather than on the weights 
themselves, which guarantee stable WTA behavior. The plastic network we present is 
able to adapt to changing input conditions, and to dynamically adjust its gain, therefore 
exhibiting self-stabilization mechanisms that are crucial for maintaining stable operation in 
large networks of interconnected subunits. We show how distributed neural assemblies 
can adjust their parameters for stable WTA function autonomously while respecting 
anatomical constraints on neural wiring. 



Keywords: winner-take-all, competition, plasticity, self-organization, contraction theory, canonical microcircuits, 
inhibitory plasticity 



1. INTRODUCTION 

Competition through shared inhibition is a powerful model of 
neural computation (Maass, 2000; Douglas and Martin, 2007). 
Competitive networks are typically composed of populations of 
excitatory neurons driving a common set of inhibitory neurons, 
which in turn provide global negative feedback to the excita- 
tory neurons (Amari and Arbib, 1977; Douglas and Martin, 1991; 
Hertz et al, 1991; Coultrip et al, 1992; Douglas et al, 1995; 
Hahnloser et al, 2000; Maass, 2000; Rabinovich et al, 2000; 
Yuille and Geiger, 2003; Rutishauser et al., 2011). Winner-take- 
aU (WTA) networks are one instance of this circuit motif, which 
has been studied extensively. Neurophysiological and anatomical 
studies have shown that WTA circuits model essential features 
of cortical networks (Douglas et al., 1989; Mountcastle, 1997; 
Binzegger et al, 2004; Douglas and Martin, 2004; Carandini 
and Heeger, 2012). An individual WTA circuit can implement a 
variety of non-linear operations such as signal restoration, ampli- 
fication, filtering, or max-like winner selection, e.g., for decision 
making (Hahnloser et al., 1999; Maass, 2000; Yuille and Geiger, 
2003; Douglas and Martin, 2007). The circuit plays an essential 
role in both early and recent models of unsupervised learning, 
such as receptive field development (von der Malsburg, 1973; 
Fukushima, 1980; Ben-Yishai et al, 1995), or map formation 
(Willshaw and Von Der Malsburg, 1976; Amari, 1980; Kohonen, 
1982; Song and Abbott, 2001). Multiple WTA instances can be 



combined to implement more powerful computations that can- 
not be achieved with a single instance, such as state dependent 
processing (Rutishauser and Douglas, 2009; Neftci et al., 2013). 
This modularity has given rise to the idea of WTA circuits rep- 
resenting canonical microcircuits, which are repeated many times 
throughout cortex and are modified slightly and combined in 
different ways to implement different functions (Douglas and 
Martin, 1991, 2004; Rutishauser et al, 2011). 

In most models of WTA circuits the network connectivity is 
given a priori. In turn, little is known about whether and how 
such connectivity could emerge without precise pre-specification. 
In this article we derive analytical constraints under which local 
synaptic plasticity on all connections of the network tunes the 
weights for WTA-type behavior. This is challenging as high-gain 
WTA operation on the one hand, and stable network dynamics on 
the other hand, impose diverging constraints on the connection 
strengths (Rutishauser et al., 201 1), which should not be violated 
by the plasticity mechanism. Previous models like lug et al. (2012) 
or Bauer (2013) have shown empirically that functional WTA-like 
behavior can arise from an interplay of plasticity on excitatory 
synapses and homeostatic mechanisms. Here, we provide a math- 
ematical explanation for this phenomenon, using a mean-field 
based analysis, and derive conditions under which biologically 
plausible plasticity rules applied to all connections of a network 
of randomly connected inhibitory and excitatory units produce 
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a functional WTA network with structured connectivity. Due to 
plastic inhibitory synapses, convergence of the model does not 
rely on constant, pre-defined inhibitory weights or other com- 
mon assumptions for WTA models. We prove that the resulting 
WTA circuits obey stability conditions imposed by contraction 
analysis (Lohmiller and Slotine, 1998; Rutishauser et al., 2011). 
This has important implications for the stability of larger net- 
works composed of multiple interconnected WTA circuits, and 
thus sheds light onto the mechanisms responsible for the emer- 
gence of both local functional cortical microcircuits and larger 
distributed coupled WTA networks. 

This article is structured as follows: We first define the network 
and plasticity models in sections 2.1 to 2.3. Our main analytical 
results are given in sections 2.4 and 2.5, and illustrated with simu- 
lation results in section 2.6. The results are discussed in section 3, 
and detailed derivations of the analytical results can be found in 
section 4. 

2. RESULTS 

2.1. NETWORK TOPOLOGY 

In its simplest abstract form, a WTA circuit (Figure lA) con- 
sists of a number of excitatory units that project onto a common 
inhibitory unit. This unit, in turn, provides recurrent inhibitory 
feedback to all excitatory units. Given appropriate connection 
strengths, such inhibition makes the excitatory units compete for 
activation in the sense that the unit receiving the strongest input 
signal will suppress the activation of all other units through the 
inhibitory feedback loop, and "win" the competition. 

We design a biologically plausible network by taking into 
account that inhibitory feedback is local, i.e., it only affects cells 
within a cortical volume that is small enough such that the rel- 
atively short inhibitory axonal arbors can reach their targets. We 
assume excitatory and inhibitory neurons in this volume to be 
connected randomly (see Figure IB). Furthermore, we assume 
that there are a finite number of different input signals, each acti- 
vating a subset of the excitatory cells in the volume. We construct 
a mean-field model by grouping the excitatory neurons for each 
driving input stimulus, summarizing the activity of each group 
of cells by their average firing rate. This results in a simplified 
population model of the network which — in the case of two dif- 
ferent input signals — consists of two excitatory populations (one 
for each input), and one inhibitory population (see Figure IC). 
We assume fuU recurrent connectivity between all populations. 
This scheme can easily be extended toward more input groups. In 
particular, if an excitatory group receives multiple inputs, it can 
be modeled as a new class. 

Since inhibitory axons are (typically) short-range, distant 
populations can communicate only via excitatory projections. 
We combine multiple local circuits of the form shown in 
Figures 1B,C by introducing excitatory long-range connections 
between them, as illustrated in Figure ID. Specifically, we add 
projections from the excitatory populations of one local group 
to all excitatory and inhibitory populations of the other group. A 
similar connectivity scheme for implementing distributed WTA 
networks has been proposed by Rutishauser et al. (2012). Unlike 
their model, our network does not require specific wiring, but 
rather targets any potential cell in the other volume. We will 



A 
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FIGURE 1 I Illustration of the network model. (A) Abstract 
representation of a WTA circuit, where several excitatory units project onto 
a common inhibitory unit, and receive global inhibitory feedback from that 
unit. (B) Example volume of the generic cortical structure that is assumed, 
consisting of (initially randomly connected) excitatory (pyramidal) cells and 
inhibitory interneurons. The color of the cells indicates the input channel 
they are connected to: some cells only receive input from the blue, others 
from the orange source. It is assumed that the volume is sufficiently small, 
such that all excitatory cells can be reached by the (short-ranged) axons of 
the inhibitory cells. If the connection strengths are tuned appropriately, the 
population receiving the stronger input signal will suppress the response of 
the weaker population via the global inhibitory feedback. (C) shows the 
mean field model of the same network that we construct by grouping 
excitatory neurons by their input source. The three resulting excitatory and 
interneuron populations are connected in an all-to-all fashion. (D,E) show 
multiple, distant volumes which are connected via long-range excitatory 
connections. Projections from one volume to another connect to all cells of 
the target volume. In (E), the two subgroups are approximated by networks 
of the type shown in (C), consisting of one inhibitory and several excitatory 
populations. The black, solid arrows represent exemplary excitatory 
connections from one population of one group to all populations of the 
other group. Equivalent connections, indicated by dotted arrows, exist for 
all of the excitatory populations. 



show in section 2.4.4 that this is sufficient to achieve competition 
between units of spatially distributed WTA circuits. 

2.2. NETWORK DYNAMICS 

The activation of a neural populations x,-, which can be excitatory 
or inhibitory, is described by 



r,-x,(f) = -x,(f) -I- 



^ WyX,(f) -I- /ext,!(f) - T, 



(1) 



where r, is the time constant of the population, Wy is the weight 
of the incoming connection from the ;th population, /ext,i(f) is 
an external input given to the population, and T, is the activation 
threshold. Furthermore, [v]-|- := max(0, v) is a half-wave rectifi- 
cation function, preventing the firing rates from taking negative 
values. Assuming identical time constants for all populations, i.e., 
r," = T for all i, the dynamics of the fuU system can be written as 
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rx(f) = -x{t) + [Wx{t) + /ext(f) - T]+ , (2) 

where x = {xi, . . . , xm) SlK the firing rates of the respective pop- 
ulations (excitatory and inhibitory), W is the connectivity matrix 
(describing local excitatory, local inhibitory, and long-range exci- 
tatory connections), /ext(f) is a vector of external inputs, and 
T = {Ti, . . . , Tp]) are the activation thresholds of the popula- 
tions. For the single local microcircuit shown in Figure IC, for 
example, W would be a 3-by-3 matrix with all entries Wy non- 
zero except for the inhibitory to inhibitory coupling. For two 
coupled microcircuits as in Figure IE, the connectivity matrix 
consists of 4 blocks, with the diagonal blocks describing local 
connectivity, and the off-diagonal blocks describing long-range 
projections from excitatory units to the other circuit. 

2.3. PLASTICITY MECHANISMS AND WEIGHT DYNAMICS 

In our model, we assume that all connections Wy in Equation (2) 
are plastic, and are subject to the following weight update rule: 

W = Tjf XpreXpost (:X:post(Wn,ax - w) - i&w + A„Xp,e)w) . (3) 

Here, Xp^e and Xpost are the pre- and postsynaptic firing rates, 
respectively, Wmax is the maximum possible weight value, and &„, 
Aw, and ts are positive constants, which we set to values that are 
compatible with experimental findings (see Table 1). The learning 
rate is determined by Tj, and and A„ determine the point at 
which the rule switches between depression (LTD) and potentia- 
tion (LTP). We will show that in a plastic network, global stability 
and circuit function are determined exclusively by those plastic- 
ity parameters. The plasticity rule is derived from the mean-field 
approximation of the triplet STDP rule by Pfister and Gerstner 
(2006), which we augment with a weight-dependent term, effec- 
tively limiting the weight values to the interval [0, Wmaxl- A more 
detailed derivation of the learning rule can be found in the 
Methods (section 4.1). The parameters &„ and A„ are set dif- 
ferently for excitatory and inhibitory connections, leading to two 
types of simultaneously active plasticity mechanisms and weight 
dynamics, even though the same learning equation is used. We 
set = ©exc and A-^, = Agxc for all excitatory connections, and 
©jy = ©ijjjj and A„ = Ai^h for all inhibitory connections. In par- 
ticular, we assume Ainh to take very low values and set Ai^h = 0 in 
our analysis, effectively eliminating any dependence of the fixed 
point of inhibitory weights on the presynaptic rate. According to 
fits of the parameters to experimental data (see Table 1), this is a 
plausible assumption. For the sake of simplicity, we also assume 



the maximum possible weight value Wmax to be the same for 
all excitatory and inhibitory connections. Figure 2 illustrates the 
weight change as a function of the pre- and postsynaptic activity. 

2.4. STABILITY ANALYSIS 

The WTA circuit is assumed to function correctly if it converges 
to a stable state that represents the outcome of the computation it 
is supposed to perform. Conditions under which these networks 
converge to their (single) attractor state exponentially fast were 
previously derived by Rutishauser et al. (2011). Here, we extend 
those results to plastic networks and express stability criteria in 
terms of global learning rule parameters, rather than individual 
weight values. We first describe criteria for the stabilization of the 
network and learning rule dynamics, then derive from them con- 
ditions on the learning rule parameters. Our analysis leads to very 
simple sufficient conditions that ensure the desired stable WTA 
behavior. 

The dynamics of the network activation and the weights are 
given by Equations (2) and (3), respectively. In the following, we 
will denote them by / and g, so the fuU dynamics can be written 
as a coupled dynamical system 

x=f{x,w), (4) 
w = g(x, w), (5) 

where / corresponds to the right hand side of Equation (2), and 
g combines the update rules for all weights (with different sets 
of parameters for excitatory and inhibitory connections) in one 
vector-valued function. We first restrict our analysis to the sim- 
plest case of a single winning excitatory population and derive 
conditions under which the plastic network converges to its fixed 
point. Later, we extend our analysis to larger systems of multiple 
coupled excitatory populations. 

2.4. 1. Analysis of single-node system 

Let us first consider a simplified system, in which only one exci- 
tatory population is active, e.g., because one population receives 
much more external input than all others, and the inhibitory 
feedback suppresses the other populations. As silent populations 
neither contribute to the network dynamics nor to the weight 
dynamics, they can be excluded from the analysis. We can there- 
fore reduce the description of the system to a single excitatory 
population xg, and an inhibitory population xi, together with the 
connections we ^ Ei we ^ !> and wj ^ e between them. 



Table 1 | Learning rule parameters /tj , 


and from Pfister and Gerstner (2006). 
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A+ 




A2 A3 
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A 




All-to-all 


5 X 10-10 


6.2 X 10-3 


7 X 10-3 2.3 X 10-'* 


18.19 


0.06 


1.3 X 10-5 


Nearest spike 


8.8 X IQ-''^ 


5.3 X 10-2 


6.6 X 10-3 2A X 10-3 


6.24 


2.09 


3.6 X 10-5 



The values of 8, A, and ts for the plasticity rule Equation (3) have been computed using Equations (14) to (16). The data corresponds to fits of the triplet Spike- 
Timing Dependent Plasticity (STDP) model with all-to-all spike interactions (first row) and with nearest spike interactions (second row) to recordings from plasticity 
experiments in rat visual cortex. Note that the time constants zx.y and t± are not reproduced here. However, they all are of the order of hundreds of milliseconds 
and can be found in Pfister and Gerstner (2008). In our simulations, we use parameters very similar to the "all-to-all" parameters for inhibitory connections, while 
for excitatory connections we use ones that are close to the "nearest spike " parameters. 
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For a given set of (fixed) weights Wc, Rutishauser et al. (201 1) 
have shown by means of contraction theory (Lohmiller and 
Slotine, 1998) that the system of network activations x = f{x, Wc) 
converges to its fixed point x* exponentially fast if its generalized 
Jacobian is negative definite. In our case, this condition reduces to 

Re(wE^E - 2 + {wl^^ - iwi^EWE^iY^^^ < 0. (6) 

If condition (6) is met, the system is called contracting and is 
guaranteed to converge to its attractor state 

Xl = A/ext, (7) 
Xj* = AWE^l/ext, (8) 

exponentially fast for any constant input /ext> where the contrac- 
tion rate is given by the left hand side of (6), divided by 2t. Here, 
A = (1 — we^e + we^i wi^e)^^ corresponds to the network 
gain. A more detailed derivation of the fixed point can be found 
in section 4.2. Note that we have set the activation threshold T 
equal to zero and provide external input /ext to the excitatory pop- 
ulation only. This simplifies the analysis but does not affect our 
results qualitatively. 

2.4.2. Decoupling of network and weight dynamics 

In the following, we assume that the population dynamics is 
contracting, i.e., that condition (6) is met, to show that the plas- 
ticity dynamics Equation (5) drives the weights w to a state that 
is consistent with this condition. Essentially, our analysis has to 
be self-consistent with respect to the contraction of the activa- 
tion dynamics. If we assume / and g to operate on very different 
timescales, we can decouple the two systems given by Equations 
(4) and (5). This is a valid assumption since neural (population) 
dynamics vary on timescales of tens or hundreds of milliseconds 



(see Figures for typical timescales of our system), while synap- 
tic plasticity typically acts on timescales of seconds or minutes. 
This means that from the point of view of the weight dynam- 
ics g the population activation is at its fixed point x* almost all 
the time, because it converges to that point exponentially fast. We 
can thus model the activation dynamics as a quasi-static system, 
and approximate the learning dynamics as a function of the fixed 
point of the activation instead of the instantaneous activation. 

g{x,w)^ g(x*,w), (9) 

The fixed point of this simplified system is found by setting 
g{x* , w) = 0, and according to Equation (3) is given by 

w*= ^^ , . (10) 

0VV ~t~ ^w-'^-pj-e ~t~ •''■post 

Combining this expression with Equations (7) and (8) leads 
to a system of non-linear equations that can be solved for the 
fixed point weights ^ £, w| ^ j, wf_^ j, and activations x^, and 
X*. These values solely depend on the learning rule parameters 
&„, A„, Wmaxi and the external (training) input /ext- 

Figure 3 shows the fixed points of the weight dynamics as a 
function of Gexo and the input strength /ext. Notably, w^^e ^^'^ 

_^ J lie on a fixed line in the we ^ e - we ^ I plane for all parame- 
ters ©w and A„. As the weight values are bounded by 0 and Wmax, 
the weights converge to a finite value for 7ext ^ oo. This is also 
illustrated in Figure 4, which shows the final weight values as a 
function of w^axy both for a finite training input and in the limit 

/ext OO. 

Importantly, the function of a WTA circuit critically depends 
on the strength of the recurrent connection we^e (Rutishauser 
et al, 201 1). If we^ E > !> the network operates in "hard" mode, 
where only one unit can win at a time and the activation of all 



— 15 Hz 35 Hz 

— 25 Hz — - Inh. rule 

(25 Hz) 




0 10 20 30 0 10 20 30 

-''post 

(Hz) Xpre (Hz) 



FIGURE 2 I Illustration of the learning rule. Tine weiglit cliange dwjdt is 
plotted as a function of tfie post- and presynaptic firing rate for fixed pre- (left) 
or postsynaptic (right) rates. The gray, dashed line shows the rule that we use 
for inhibitory connections and whose threshold for LTR in contrast to 
excitatory connections, does not depend on the presynaptic rate. The black 
line marks the transition between LTD and LTR In this example, the 
parameters of the learning rules were set to 9exc = 6 Hz, C-)|nh = 18 Hz, 
/\exc = 2, Wmax = 4, and the weight value was fixed at Wmax/3 for excitatory 
and Wmax/2 for inhibitory connections. 




0.25 0.4 0.55 0.22 0.26 0.3 

FIGURE 3 I Illustration of the fixed point in weight space. The va ues of 

the final weights (in units of Wmax) are plotted as functions of the parameter 
©exc and the training input strength /ext, where bigger circles correspond to 
greater /ext. The left panel shows the g plane, while the right 

panel shows the |-we^ e plane. Interestingly, the fixed point in 
We^ i-We^ E space only gets shifted along a line for different values of 
©exc and /ext- For /ext — >• CO the weights converge to a limit point, as is 
illustrated in Figure 4. For these plots, the parameters Aexc and 9|nh were 
set to 2 and 18 Hz, respectively, and Wmax was set to 4. 
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Other units is zero. On the other hand, if we ^ e is smaller than 
1, the network implements "soft" competition, which means that 
multiple units can be active at the same time. From Equation (27) 
(Methods) it follows that we^e > 1 is possible only if Wmax > 
A + 1. As we will show in the following section, this condition 
is necessarily satisfied by learning rules that lead to stable WTA 
circuits. 



Note that condition ( 1 1 ) is independent of ©exc and ©inh . This 
is due to a simplification that is based on the assumption Aexc + 
b ^ I, which can be made without loss of generality. If fo and Aexc 
are set to very low values, the full expressions given by 38 and 
(39) (see Methods section 4.2) apply instead. Figure 5 shows the 
the region defined by ( 1 1 ) for different b together with the exact 



2.4.3. Parameter regimes for stable network function 

We can now use the fixed points found in the previous section 
to express the condition for contraction given by condition (6) in 
terms of the learning rule parameters. In general, this new con- 
dition does not assume an analytically simple form. However, we 
can find simple sufficient conditions which still provide a good 
approximation to the actual value (see Methods section 4.2 for 
details). Specifically, as a key result of our analysis we derive the 
following sufficient condition: Convergence to a point in weight 
space that produces stable network dynamics is guaranteed if 



10 20 30 40 



+ b < Wmax < 2(1 + A 

exc/ 5 



(11) 



where b is a parameter of the order 1, which is related to the mini- 
mum activation xe (or the minimum non-zero input 7ext) during 
training for which this condition should hold. If the minimum 
input /mil) that the network will be trained on is known, then b can 



be computed from the fixed point ^ 



i), and 



set to b = ©exc/^E min' '^^^^ ^iU guarantee contracting dynam- 
ics for the fuU range of training inputs 7ext € [/mim oo). In typical 
scenarios, b can be set to a number of the order 1 . This is due to 
the fact that the network activation is roughly of the same order 
as the input strength. Setting ©exc to a value of similar order leads 

tob = ©exc/*J ■ ~ 1- 
E,min 
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FIGURE 4 1 Limit beliavior of the fixed point of tfie weights for weal< 


and strong inputs. Tine final weight values (in units of Wmax) are plotted as 


a function of Wmax 


both for /ext = 15 Hz (solid lines) and in the limit of very 


large inputs /ext 


cxD (dashed lines). 


In the limit case, e and e 


converge to expressions that are linear in Wmax, while i increases 


superlinearly. The learning rule parameters were set to ©exc = 6 Hz, 


Binh = 18 Hz, and/\exc = 2. 








FIGURE 5 I Regions in learning rule parameter space that lead to 
a stable, contracting network. All panels show the regions of 
stability in Wmax-Aexc space for different training input strengths. 
Colored lines correspond to exact solutions, while black, dotted lines 
correspond to the sufficient condition (11) for different values of fa. 
The top panels illustrate that relatively small values of fa (e.g., 2) 
roughly approximate the exact solution even for very small inputs 
(e.g., /ext = 1 Hz; left), whereas fa can be set to lower values (e.g., 
fa = 0; right) if the input is larger. The gray-scale value represents the 
convergence rate |^| (in units of s^M of the activation dynamics for 
r = 10ms. The bottom panel shows in color the exact regions of 
contraction for inputs /ext = 1 , 10, 100 Hz and the approximation given 
by condition (11) for fa= 0,1,2. Some of the colored regions (and 
dotted lines) correspond to the ones shown in the upper panels. It 
can be seen that for higher input strengths the upper bound on 
Aexc (or equivalently, the lower bound on Wmax) quickly converges to 
the fa = 0 diagonal, which represents the asymptotic condition for 
/exc -* 00. For these plots, the learning rule parameters were set to 
eexc = 6Hz and 0inh = 18Hz. 
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condition for contraction, indicating that ( 1 1 ) is indeed sufficient 
and that b can safely be set to a value around 1 in most cases. 

2.4.4. Extension to multiple units 

So far, we have only studied a small network that can be regarded 
as a single subunit of a larger, distributed WTA system. However, 
our results can be generalized to larger systems without much 
effort. In our model, as illustrated in Figures 1D,E, different local- 
ized WTA circuits can be coupled via excitatory projections. These 
projections include excitatory-to-inhibitory connections, as well 
as reciprocal connections between distant excitatory units. In 
order to demonstrate the effects of this coupling, we consider 
two localized subsystems, X = {xe, xi)andx' = (xg, xj), consist- 
ing of one excitatory and one inhibitory unit each. Furthermore, 
we add projections from to x'^ and x[, as required by our 
model. We denote by We^e' the strength of the long-range 
excitatory-to-excitatory connection, while we refer to the long- 
range excitatory- to-inhibitory connection as w^^y. Note that 
for the sake of clarity we only consider the unidirectional case 
X ^ x' here, while the symmetric case x -o- a/ can be dealt with 
analogously. 

We first look at the excitatory-to-inhibitory connections. If 
only Xe is active and x'^ is silent, then xi and x[ are driven by 
the same presynaptic population (xe), and we^v converges to 
the same value as we ^ i. Thus, after convergence, both inhibitory 
units are perfectly synchronized in their activation when x^ is 
active, and an equal amount of inhibition can be provided to xe 

Besides synchronization of inhibition, proper WTA function- 
ality also requires the recurrent excitation e' (between the 
excitatory populations of the different subunits) to converge 
to sufficiently low values, such that different units compete 
via the synchronized inhibition rather than exciting each other 
through the excitatory links. As pointed out by Rutishauser et al. 
(2012), the network is stable and functions correctly if the recur- 
rent excitation between populations is lower than the recurrent 

self-excitation, i.e., We ^e' < we'^e'- 

We now consider the case where xe and x'^ receive an exter- 
nal input /ext. Whenever alone receives the input, there is 
no interaction between the two subunits, and the recurrent self- 
connection We' E' converges to the value that was found for 
the simplified case of a single subunit (section 2.4.2). The same 
is true for the connection we ^ e if *E alone receives the input. 
However, in this case xe and Xg might also interact via the con- 
nection we^e'j which would then be subject to plasticity. As x 
projects to x', but not vice versa, we require xe > Xg if both xe 
and Xg receive the same input 7ext, because xe should suppress Xg 
via the long-range competition mechanism. In terms of connec- 
tion strengths, this means that Wg _^ j, w*, ^ g, > Wg ^ g,, i.e., the 
inhibitory input to x^ that is due to xe must be greater than the 
excitatory input Xg receives from xe. In the Methods (section 4.3), 
we show that a sufficient condition for this to be the case is 

Wmax>^ + fc+l, (12) 

which alters our results from section 2.4.3 only slightly, effec- 
tively shifting the lower bound on Wmax by an offset of 1, as can 



be seen by comparing conditions (11) and (12). On the other 
hand, making use of the fact that Xg < xe, it can be shown that 
we^E' converges to a value smaller than we'^e' (see Methods 
section 4.3), as required by the stability condition mentioned 
above. 

2.5. GAIN CONTROL AND NORMALIZATION 

In the previous section, we showed how synaptic plasticity can 
be used to drive the connection strengths toward regimes which 
guarantee stable network dynamics. Since the actual fixed point 
values of the weights change with the training input, this mech- 
anism can as well be used to tune certain functional properties 
of the network. Here we focus on controlling the gain of the net- 
work, i.e., the relationship between the strength of the strongest 
input and the activation of the winning excitatory units within 
the recurrent circuit, as a function of the training input. 

In the case of a single active population, the gain is given by 
A = XE//ext = (1 — ^E^E + Wg^ J Wj%g)^^ as can be inferred 
from Equation (7). Depending on the gain, the network can either 
amplify (A > 1) or weaken (A < 1) the input signal. 

Figure 6 shows how the gain varies as a function of the learning 
rule parameters and the training input strength 7ext- Low aver- 
age input strengths cause the weights to converge to values that 
lead to an increased gain, while higher training inputs lower the 
gain. This can be regarded as a homeostatic mechanism, acting to 
keep the network output within a preferred range. This provides 
a mechanism for the network to adapt to a wide range of input 
strengths, while stiU allowing stable WTA competition. 

2.6. SIMULATION RESULTS 

As a final step, we verify the analytical results in software sim- 
ulations of a distributed, plastic WTA network, as illustrated in 
Figures 1D,E. Note that here we consider the case where two 
subgroups are coupled bidirectionally via excitatory long-range 
projections, while in section 2.4.4, for the sake of clarity, we 
focus on the unidirectional case. The desired functionality of 
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the resulting network is global competition between the excita- 
tory populations, i.e., the population that receives the strongest 
input should suppress activation of the other populations, even 
if the excitatory populations are not directly competing via the 
same, local inhibitory population. We consider a network with 
two groups, each consisting of two excitatory populations and one 
inhibitory population (see Figure IE). While the excitatory pop- 
ulations are connected in an all-to-all manner, inhibitory pop- 
ulations can only target the excitatory populations within their 
local groups, but do not form long-range projection. Initially, all 
connection weights (excitatory and inhibitory) are set to random 
values between 0.3 and 1.8. Note that those values could poten- 
tially violate the conditions for contraction defined in (6), but we 
will show empirically that the plasticity mechanism can stiU drive 
the weights toward stable regimes. As training input, we present 
1000 constant patterns for 2 s each. In every step, four input values 
in the ranges 5 ± 2 Hz, 10 ± 2 Hz, 15 ± 2 Hz, and 20 ± 2 Hz are 
drawn from uniform distributions and applied to the four exci- 
tatory units. The different input signals are randomly assigned to 
the populations in every step, such that a randomly chosen popu- 
lation receives the strongest input. Thereby, each population only 
receives one of the four inputs. 

Figure 7 shows the activation of the different populations 
before and after learning. Before learning (left), the network does 
not necessarily implement stable competition between the differ- 
ent excitatory populations. Instead, it may end up in an oscillating 
state or amplify the wrong winning unit. However, after training 
(Figure 7, right), the network always converges to a stable state 
representing the winner of the competition. Furthermore, it can 



FIGURE 7 I Simulated evolution of the plastic network. 1000 input 
patterns were applied for 2 s each. Two local subsystems, consisting of two 
excitatory units and one inhibitory unit each, are coupled via all-to-al 
excitatory connections, while inhibitory feedback is provided only locally. The 
first three rows show the populations of the first group, where the first two 
rows correspond to the two excitatory populations, and the third row shows 
the activation of the inhibitory population. The last three rows show the 
activations of the second group. The left panel shows the first 20 s after 
initialization (before learning), while the right panel shows the network 
activity during the last 20 s (after learning). Solid blue and orange lines 
correspond to the firing rate of the respective population, whereby the 



be seen that the inhibitory populations perfectly synchronize, as 
described in section 2.4.4. 

The change of weights is illustrated in Figure 8: Initially (top), 
all weights were set to random values in the range [0.3, 1.8]. 
Since all populations receive the same average input, the weight 
matrices should converge to symmetric states. For the specific 
set of learning rule parameters we chose in this example, and 
the specific input rates described above, we ^ e converges to a 
value around 1, which means that the network is at the edge of 
the transition between hard and soft WTA behavior. The weights 
We ^ I, connecting excitatory to inhibitory units, converge to val- 
ues around 2. Furthermore, the weights wi^Ej which connect 
inhibitory to excitatory units all converge to very similar val- 
ues (around 1.1), such that inhibition is synchronized across the 
whole network. Note that not all connections between excitatory 
populations have converged to the same value. This is because as 
soon as the network is close to the hard WTA regime, some con- 
nections cannot change anymore as only one excitatory unit is 
active at a time, and the weight change is zero if either the pre- or 
the post-synaptic unit is inactive. 

3. DISCUSSION 

We have shown how neural circuits of excitatory and inhibitory 
neurons can self-organize to implement stable WTA competi- 
tion. This is achieved through an interplay of excitatory and 
inhibitory plasticity mechanisms operating on all synapses of the 
network. As a key result, we provide analytical constraints on 
the learning rule parameters, which guarantee emergence of the 
desired network function. 
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highlighted segments (orange) mark the winner among the four excitatory 
populations. The input given to the individual units is plotted as a dotted black 
or solid magenta line, where the highlighted segments (magenta) correspond 
to the strongest signal among the four. Thus, the network operates correctly 
if magenta and orange lines are aligned (the one population that receives the 
strongest input wins the competition), while misaligned lines (the population 
receiving the strongest input does not win the competition) indicate incorrect 
operation. Initially, the network frequently selects the wrong winning unit and 
even starts oscillating for some input patterns (around 18 s). After training 
(right), the network converges to a stable state with only the winning unit 
active for different input patterns. 
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FIGURE 8 I Evolution of the connectivity matrices. Tlie top panels show 
the (initially random) weight matrices, and the lower panels show the final 
weight matrices after 2000 s of simulated training time. The left panel shows 
the connections between excitatory populations. The panel in the middle 
represents the connections from inhibitory to excitatory populations. Note 
that some inhibitory connections are set to zero as inhibitory units can only 
project to targets within their own local group, according to our model. The 
right panel shows the connections from excitatory to inhibitory units. Here, 
connections from the same source converge to the same value, leading to 
perfect synchronization of the two inhibitory units. The (rounded) weight 
values are displayed on top of the image. 



Although constraints on the weights for stable competition in 
recurrent excitatory-inhibitory circuits have been derived before 
(Xie et al., 2002; Hahnloser et al., 2003; Rutishauser and Douglas, 
2009; Rutishauser et al., 2012), it has remained unclear how 
a network can self-tune its synaptic weights to comply with 
these conditions. The presented model achieves this and pro- 
vides important insights regarding the mechanisms responsible 
for this self-tuning. Our results predict a relationship between 
the maximum synaptic weight Wmax in a circuit and the learn- 
ing rule parameter Aexc, which controls the contribution of 
the presynaptic rate to the shifting of the threshold between 
potentiation and depression. Furthermore, our model predicts a 
relationship between the network gain and the amount of excita- 
tory input into the circuit during development or training (see 
Figure 6), indicating that high gain (amplification) should be 
expected for weak inputs, and low gain for strong inputs, which 
is in accordance with common assumptions about homeostasic 
mechanisms (Turrigiano, 2011). 

From a developmental perspective, the self-configuration of 
functional WTA circuits through plasticity has the advantage of 
requiring a smaller number of parameters to be encoded geneti- 
cally to obtain stable and functional network structures (Zubler 
et al., 2013). With self-tuning mechanisms like the ones sug- 
gested here, only the parameters for the two different types of 
plasticity in excitatory and inhibitory synapses, rather than the 



strengths of all synaptic connections, need to be specified, and 
the network can adapt to the statistics of inputs it receives from 
its environment and from other brain regions. 

Besides guaranteeing stability, it is also desirable to control 
functional properties of the circuit, such as its gain. Experimental 
data suggests that cortical recurrent circuits often operate in a 
high gain regime and with strong (larger than unity) recurrent 
excitatory feedback (Douglas et al., 1995). The strength of this 
feedback determines whether the WTA is "soft" (multiple excita- 
tory units can be active at the same time) or "hard" (only one unit 
can be active at a time, i.e., the network operates in a nonlinear 
regime) (Rutishauser et al., 2011). Many interesting computa- 
tions that can be realized with these types of networks rely on 
the non-linearities introduced by such strong recurrent excita- 
tion (e.g., Vapnik, 2000), therefore it is important that similar 
conditions can be achieved with our model. In addition, various 
forms of learning rely on balanced WTA competition (Masquelier 
et al, 2009; Habenschuss et al, 2012; Nessler et al, 2013), which 
requires an adaptation of the gain as the excitatory connections 
into the circuit undergo plasticity. In our network, the resulting 
network gain is a function of both the learning rule parameters 
and the strength of the training input signals. As a consequence, 
our system can switch between high and low gain, and hard 
or soft WTA behavior simply by receiving input stimuli of dif- 
ferent (average) strengths. Thus, different parts of the network 
might develop into different functional modules, depending on 
the inputs they receive. 

Our model does not specifically address the question of how 
the network structure, which leads to our results (essentially ran- 
dom all-to-all connectivity) might develop in the first place. For 
instance, if certain long-range connections between multiple sub- 
circuits do not exist initially, they will never be established by 
our model, and the units of the different subcircuits can never 
compete. On the one hand, this might be a desired effect, e.g., 
to construct hierarchies or asymmetric structures for competi- 
tion, in which some parts of the network are able to suppress 
other parts, but not vice-versa. On the other hand, structural 
plasticity could account for the creation of missing synaptic con- 
nections, or the removal of ineffective connections if the desired 
stable function cannot be achieved with the anatomical sub- 
strate. There is increasing evidence for activity dependent synapse 
formation and elimination in both juvenile and adult brains 
(Butz et al., 2009), in particular a coordinated restructuring of 
inhibitory and excitatory synapses for functional reorganization 
(Chen and Nedivi, 2013). Another approach, recently investigated 
in simulations by Bauer (2013), is to set up the right network 
topology by developmental self-construction processes in a first 
step, and the tune the network using synaptic plasticity in a 
second step. 

Our model is based on a weight-dependent variation of the 
learning rule proposed by Pfister and Gerstner (2006), but this is 
by no means the only learning rule capable of the self-calibration 
effect we describe in this article. By changing its parametrization, 
the rule can subsume a wide variety of commonly used Hebbian, 
STDP-like, and homeostatic plasticity mechanisms. Indeed, fur- 
ther experiments, which are not presented in this manuscript, 
indicate that a whole class of learning rules with depression at 
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low and potentiation at high postsynaptic firing rates would lead 
to similar results. We chose the triplet rule to demonstrate our 
findings as its parameters have been mapped to experiments, 
and also because it can be written in an analytically tractable 
form. We have assumed here a specific type of inhibitory plas- 
ticity, which analytically is of the same form as the simultane- 
ous excitatory plasticity, but uses different parameters. With the 
parameters we chose for the inhibitory plasticity rule, we obtain 
a form that is very similar to the one proposed by Vogels et al. 
(20 11). By introducing inhibitory plasticity it is no longer neces- 
sary to make common but biologically unrealistic assumptions, 
like pre-specified constant and uniform inhibitory connection 
strengths (Oster et al, 2009), or more abstract forms of sum- 
ming up the excitatory activity in the circuit (lug et al., 2012; 
Nessler et al, 2013), because inhibitory weights wUl automat- 
ically converge toward stable regions. Inhibitory plasticity has 
received more attention recently with the introduction of new 
measurement techniques, and has revealed a great diversity of 
plasticity mechanisms, in line with the diversity of inhibitory 
cell types (KuUmann and Lamsa, 2011; KuUmann et al., 2012). 
Our model involves only a single inhibitory population per local 
sub-circuit, which interacts with all local excitatory units. Not 
only is this a common assumption in most previous models, 
and greatly simplifies the analysis, but also is in accordance with 
anatomical and electrophysiological results of relatively unspe- 
cific inhibitory activity in sensory cortical areas (Kerlin et al., 
2010; Bock et al, 201 1). However, recent studies have shown more 
complex interactions of different inhibitory cell types (Pfeffer 
et al., 2013), making models based on diverse cell types with 
different properties an intriguing target for future studies. The 
assumption of a common inhibitory pool that connects to all exci- 
tatory units is justified for local circuits, but violates anatomical 
constraints on the length of inhibitory axons if interacting pop- 
ulations are far apart (Binzegger et al, 2005). Our results easily 
generalize to the case of distributed inhibition, by adapting the 
model of Rutishauser et al. (2012) (see Figure IE). Our contri- 
bution is to provide the first learning theory for these types of 
circuits. 

Since our model is purely rate-based, a logical next step is 
to investigate how it translates into the spiking neural network 
domain. Establishing similar constraints on spike-based learn- 
ing rules that enable stable WTA competition remains an open 
problem for future research, although Chen et al. (2013) have 
shown empirically that WTA behavior in a circuit with topo- 
logically ordered input is possible under certain restrictions on 
initial synapse strengths, and in the presence of STDP and short- 
term plasticity. Spiking WTA circuits can potentially utilize the 
richer temporal dynamics of spike trains in the sense that the 
order of spikes and spike-spike correlations have an effect on the 
connectivity. 

Potential practical applications of our model, and future 
spiking extensions, lie in neuromorphic VLSI circuits, which 
have to deal with the problem of device mismatch (Indiveri 
et al., 2011), and can thus not be precisely configured a pri- 
ori. Our model could provide a means for the circuits to 
self-tune and autonomously adapt to the peculiarities of the 
hardware. 



4. MATERIALS AND METHODS 

4.1. DERIVATION OF THE PLASTICITY MECHANISM 

The learning rule given by Equation (3) is based on the triplet 
STDP rule by Pfister and Gerstner (2006). Since we are interested 
in the rate dynamics, we use the mean-field approximation of this 
rule, which is provided by the authors and leads to an expected 
weight change of 

- T^TxXpre) , (13) 

where Xpre, Xpost are the pre- and postsynaptic activations and 
Af , Af , T± , Tx^y are parameters that determine the amplitude of 
weight changes in the triplet STDP model. AH of the parameters 
are assumed to be positive. Through a substitution of constants 
given by 

:= A+T+ty, (14) 
&„:={A-r^-A+r+)/T^, (15) 
A„ := A^r^r^/r^, (16) 

the rule in Equation (13) can be written in the simpler form 

W = Tj^Xpre:»:post (^st " (Ovf + ^iv^pre)) , (17) 

where 0,^ is in units of a firing rate and A„ is a unitless constant. 
The terms in parentheses on the right of Equation (17) can be 
divided into a positive (LTP) part that depends on Xpost, and a 
negative (LTD) part that depends on Xpre. In order to constrain 
the range of weights, we add weight-dependent terms m-\-(w) and 
m_ (w) to the two parts of the rule, which yields 

W = Tj^XpreXpost (Xpostm+(w) - (9^ + A„Xpre)OT_ (w)) . (18) 

Throughout this manuscript, we use a simple, linear weight 
dependence = w^ax — w and m_ = w, which effectively lim- 
its the possible values of weights to the interval [0, Wmaxl- We 
chose this form, which is described by a single parameter, for 
reasons of analytical tractabUity and because it is consistent with 
experimental findings (Giitig et al., 2003). In Pfister and Gerstner 
(2006), values for the parameters r^^y, r±, and Af^ of the rule 
Equation (13) were determined from fits to experimental mea- 
surements in pyramidal cells in visual cortex (see Table 1) and 
hippocampal cultures (Bi and Poo, 1998, 2001; Sjostrom et al, 
2001; Wang et al., 2005). We used these values to calculate plau- 
sible values for A„, and Tj using Equations (14) to (16). In 
our simulations, we use parameters very similar to the exper- 
imentally derived values in Table 1. Specifically, for inhibitory 
connections we use parameters very similar to the ones found 
from fits of experimental data to the triplet STDP model with 
all-to-all spike interactions. On the other hand, we choose param- 
eters for the excitatory plasticity rules which are close to fits of the 
triplet STDP rule with nearest-neighbor spike interactions. The 
parameters that were used in software simulations and to obtain 
most of the numeric results are listed in Table 2. Note that for the 
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Table 2 | Model parameters used in software simulation. 



Parameter 


Value 


Description 


®exc 


D nZ 


Learning rule parameter 


Oinh 


18Hz 


Learning rule parameter 




2 


Learning rule parameter 


^max 


4 


Maximum weigint value 


r2 

"■s.exc 


3.6 ms^ 


Exc. connection learning rate parameter 


s.inh 


I.Sms^ 


Inh. connection learning rate parameter 




5 ms 


Exc. population time constant 


^inh 


1 ms 


Inh. population time constant 



weight-dependent rule in Equation (18) we have assumed that the 
parameter 0,^ influences only the LTD part. According to the def- 
inition in Equation (15), this is the case if S> A^, or 0,^ ~ 
A^X-jx^, respectively. Otherwise 0„, contains both a potentiat- 
ing (A^) and a depressing (^2") component, and Equation (18) 
should be replaced with a more complex expression of the form 
of Equation (13). 



4.2. DERIVATION OF THE STABILITY CRITERIA 

In section 2.4, we outlined how the fixed points and stability cri- 
teria for the WTA system can be found. In this section, we provide 
the detailed derivations that led to these results. 

As described in section 2.4, we first consider a simplified sys- 
tem of one excitatory and one inhibitory population, xe and x\. 

They are cou- 

_ We ^ I 0 

receive external inputs -rext(f ) with weights /xe and \x\ respectively, 
and have thresholds Te, T\. Assuming that both units are active, 
i.e., their total synaptic input is larger than their thresholds, their 
dynamics are described by 



which yield an activation vector x= {x^,x\) 
pled recurrently through a weight matrix W 



Xm\ih 



-XE -I- We- 

-x\ -l- We - 



. E^E - Wi ^ 
IXe -I- Mllext 



EXi -I- /XE-fext ■ 



Te, 



(19) 
(20) 



where texc, tinh are the population time constants. The fixed 
points of the activations can be found by setting x^ = x\ = 0. If 
we assume, for simplicity, that Te = T[ = 0 this yields the fixed 
points 



A7, 
AI, 



ext (me 
ext (we - 



- Wi_ 
IME 



■EMi) , 
- (WE- 



i)w). 



where 



A = (1 - we^e + we^iwi^e) 



(21) 
(22) 



(23) 



is the network gain. Furthermore, we can make the assump- 
tion that fJ-i = 0 and /xe = 1, effectively disabling the external 
input to the inhibitory population. This reduces Equations (21) 
and (22) to 



A/ext, 

AWE^l/ext- 



(24) 
(25) 



These simplifications do not change the results of our 
analysis qualitatively and can be made without loss of 
generality. 

Approximating Xpre and Xpost by their fixed point activities 
(as described in section 2.4), and setting w = 0 in the learning 
rule Equation (18), the fixed point of the weight dynamics (with 
w > 0) takes the form 



^ost 



0V1/ -|" Ar 



-I- y"^ 
' pre ' -^post 



(26) 



Note that this fixed point in weight space always exists for any 
given Xpre and Xpost, and is stable for the weight dependence 
m-i- (w) = Wmax — WJ- (w) = w that we chose in Equation ( 18). 
In fact, this is true for all choices of the weight dependence satis- 
fying 9m-|-/9w < 0 and 9m_/9w > 0, as can be shown by means 
of a linear stability analysis. 

We now derive the fixed points for the weights we^ Ej we^ i, 
and wi^E of the simplified system. For we^E; Equation (26) 



can be simplified by noting that x*^. 



■^post 



leading to 



an expression that depends on the activation of the excitatory 
population Xg: 



We_ 



©exc/ ^E ~^ ^exc + 1 



(27) 



Similarly, we can compute the fixed point of we ^ i as a func- 



tion of Xp, noting that x. 
and (25)]: 



post 



■ WE^iXp [see Equations (24) 



©exc/x* ~ Ae: 



(28) 



Finally, using the relationship x^ = we ^ iXp from Equations 
(24) and (25), and the previously computed value of we^i 
from Equation (28) with the fixed point equation for wi^ Ej we 
obtain 



* "max , , 

Wt % F = ? ^ -(29) 

0inh/4 - ^inh (©exc/^ + (^exc " Wmax)) + 1 

In the following, we set A[nh = 0, as described in section 2.3. 
An exact solution for the activation Xp at the fixed point of 
the system is obtained by inserting Wp^p, Wp^j, and Wj*^ p 
into Equation (24), and solving the resulting fixed-point prob- 
lem Xp = /(Xp). This corresponds to finding the roots of the third 
order polynomial 



P{x) 



with coefficients 



(30) 
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flO — ©excQinh^ect; 

flj = — 0exc0inh + ©exc^ext "t" ©inh^ext "I" ©inh^exc^ext 



"I" ©exc^max! 
fl2 = -0exc - ©inh - ©inhA 

+ ©inhWinax + 20exc^excWmax 
fl3 = -1 - Aexc + Wmax + ^exc^max + ^L^max - W^ax 

- ^excl^ax- 



(31) 



(32) 



exc + ^ext + -Aexc^ext + ©exc^max 
— ©excW^ax' 



(33) 



(34) 



On the other hand, in the case of very small inputs we have to 

include the effects of b, as b can in principle take very large values. 
In typical scenarios the output of the network can be assumed 
to be roughly of the order of its input. If ©exc is chosen to be of 
the same order, then ~ 1. For any finite b, we can express the 
stability condition that is valid for all inputs as the intersection of 
the conditions for large inputs, condition (42), with the one for 
arbitrarily small inputs, condition (41), leading to 



+ b < Wmax < 2(1 + 



(43) 



The activation of the excitatory population xe at the fixed point is 
then given by the positive, real root of Equation (30). 

The fixed point of the activation Xg, and thus the fixed points 
of the weights, are monotonic functions of the training input 
strength 7ext (see Figure 3, for example). In the following, we 
investigate the behavior of the fixed point weight values for very 
large and very small external inputs during training, respectively. 
This helps us to find conditions on the learning rule param- 
eters that lead to stable dynamics (of the network activation) 
for any training input strength. We define a positive constant 
b := ©exc/^E> Pl^8 it iiito Equations (27)-(29). This yields 



Aexc + b+l 
_^ [ = Wmax ~ ^exc ~ 
Wmax© exc 



Wl^E = 



b@inh + @e 



(35) 
(36) 
(37) 



Inserting Equations (35)-(37) into the condition for contrac- 
tion of the activation dynamics given by (6), we can describe the 
condition in terms of the learning rule parameters, and a new 
constant 0 := ©exc/(©exc + ^'©mh): 



1 



{Ae^c + ^') 1 + 



(1+Aexc + fc) 
1 



< 0 < 1, 



(38) 



©(l+Aexc + ^')2-l 



Note that this condition can be met for any finite b by choos- 
ing sufficiently large Aexc and Wmax- However, as discussed above, 
choices of the parameter b of the order 1 should be sufficient for 
typical scenarios, whereas higher values would guarantee stable 
dynamics for very low input strengths (e.g., 7ext ^ ©exc)- This is 
illustrated in Figure 5, where the exact regions of stability as a 
function of Wmax and Aexc are shown for different training input 
strengths, together with the sufficient conditions given by (43). 
In practice, a good starting point for picking a value b for which 
the stability conditions should hold is to determine the minimum 
non-zero input 7min encountered during training for which this 
condition should hold, and setting b = ©exc/^^ min' ^^ere 
is the fbced point activation for 7ext = 7min- 

4.3. EXTENSION TO MULTIPLE UNITS 

In this section, we illustrate how multiple subunits, as analyzed in 
the previous section, can be combined to larger WTA networks 
with distributed inhibition. For the sake of simplicity, we only 
consider the imidirectional case, where a subunit x = (xe, xi) 
projects onto another subunit = (xg, x{) via excitatory con- 
nections We E' and we ^ r . The bidirectional case x -o- x' can 
be analyzed analogously. If xe and x^ receive the same input, the 
response of x^ should be weaker, such that activation of xe causes 
suppression of x^ rather than excitation. This means that 



Wp 



< Wc 



.I/Wj, 



(44) 



must hold. We assume that both subsystems have been trained on 
inputs of the same average strength, such that their local connec- 



< 2(1 -h Aexc + &),(39) lio^s have converged to the same weights, i.e., wt,_ 



Assuming Aexc + b^ I (note that we can always set Agxc to a 
sufficiently large value), the conditions reduce to 

0 < e < 1, (40) 

Aexc + b < Wmax < 2(1 -|- Aexc "H b), (41) 

whereby the first condition can be dropped, since 0 e [0, 1] 
always holds. The second condition still depends on b, and there- 
fore on Xg. We will illustrate how to eliminate this dependence 
under very weak assumptions. First, in the limit of very large 
inputs Xg also takes very large values, leading to ^ 0 for iext ^ 
oo. In that case, condition (41) becomes independent of b and can 
be written as 



and w*, 



£/ = w*^ g. Furthermore, we assume that condition 
(44) is true initially. This can be guaranteed by setting the ini- 
tial value of we^ E' to a sufficiendy small number. Our task then 
is to show that condition (44) remains true for all time. The 
values of w^, ^ j, and Wy^^,, or w^^j and wj'^ g respectively, 
are described by Equations (36) and (37). On the other hand, 
according to Equation (26), the value of Wg_^ g, is given by 



Wp 



^max^ 



©exc + -Aexc^ "I" ^ 



(45) 



Plugging all this into condition (44) and simplifying the expres- 
sion, leads to the condition 



Aexc < Wmax < 2(1 -f Agxc)- 



(42) 



w„ 



> ^exc + b + 



(46) 
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which can be replaced by the sufficient condition 

Wmax > Aexc + fe+ 1. (47) 

that guarantees x^, < if both excitatory populations receive 
the same input. On the other hand, this result implies 
^ g, < Wg, _^ g, , which is required for stable network dynamics 
(Rutishauser et al, 2012), and can be verified by comparing the 
respective fixed point equations 

We ^ E' = 4/ (0exc + Aexc^E + %) , (48) 
<' ^ E' = 4/ (0exc + Aexc4 + 4) • (49) 

4.4. SOFTWARE SIMULATION 

Software simulations of our model were implemented using cus- 
tom Python code based on the "NumPy" and "Dana" packages, 
and run on a Linux workstation. Numerical integration of the 
system dynamics was carried out using the forward Euler method 
with a 1 ms timestep. 
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